[Previous] [Next] [Index]
[Thread]
Re: Site Scaning & IP graps
Good spiders will ask for /robots.txt and find out what to do with
themselves if they find it.
Generally grepping for /robots.txt will give you a list of spiders that
have found you.
-brian
On Thu, 21 Mar 1996 KGANNON@dit.ie wrote:
> Date: Thu, 21 Mar 1996 13:44:27 +0000 (GMT)
> From: KGANNON@dit.ie
> To: www-security@ns2.rutgers.edu
> Subject: Site Scaning & IP graps
>
> Can anyone tell me what the best way of detecting if a spider has had a look
> at your server i.e is there a list of common spiders.
>
> The story is there is a server here on site and while looking at the logs for
> my own page I saw a number of sites that I have never heard off. The thing
> is that there is no DNS entry for the server and it has not been advertised
> only 10 or so people on site even know it exists .
>
> I am just a little curious as to how people know about the server .
>
--
Brian W. Spolarich - ANS CO+RE Systems - briansp@ans.net - (313)677-7311
Her vocabulary was as bad as, like, whatever.
Follow-Ups:
References: